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Abstract 



For developmental education students, rates of developmental math course 
completion and persistence into required college-level math courses are particularly low. 
This literature review examines the evidence base on one potential means for improving 
the course completion and learning outcomes of developmental mathematics students: 
reforming mathematics classroom pedagogy. Each study examined for this review was 
classified into one of six sets according to the main instructional approach focused on in 
the study. The six sets are: student collaboration, metacognition, problem representation, 
application, understanding student thinking, and computer-based learning. Because most 
of the studies across the sets did not employ rigorous methods, the evidence regarding the 
impact of these instructional practices on student outcomes is inconclusive. An analysis 
of the studies that did employ rigorous designs suggests that structured forms of student 
collaboration and instructional approaches that focus on problem representation may 
improve math learning and understanding. This paper concludes by making a number of 
methodological recommendations, proposing several needed areas of research related to 
developmental math pedagogy, and suggesting instructional practices that may improve 
the outcomes of developmental math students. 
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1. Introduction 



A majority of community college students enroll in developmental education 
(Bailey, Jeong, & Cho, 2010), but evidence of its effectiveness in promoting student 
progression and degree completion is mixed. While some studies have found that 
remediation reduces students’ probability of dropping out (Bettinger & Long, 2009; 
Lesik, 2007), other studies find that students in remediation accumulate fewer college 
credits and are less likely to complete a degree (Boatman & Long, 2010; Calcagno & 
Long, 2008; Martorell & McFarlin, 2008). Equally concerning are the low levels of 
developmental education course completion, especially for developmental math. At the 
57 community colleges participating in the Achieving the Dream: Community Colleges 
Count initiative (see www .achievingthedream.org) , only one third of students who were 
referred to developmental education completed the recommended sequence of math 
courses (Bailey et ah, 2010). Of the students who enrolled in developmental education 
courses, only 20% eventually completed a required college-level math course (Bailey et 
ah, 2010). 

Failing to complete developmental math and required college-level math not only 
prevents individuals from earning a college degree and pursuing certain professions but 
also has consequences for a young adult’s likelihood of employment. Young adults with 
low levels of quantitative literacy skills, including the types of arithmetic operations and 
applications typically covered in developmental math courses, are more likely to be 
unemployed — and, moreover, low levels of quantitative literacy partially account for the 
lower employment rates for African Americans compared to Whites (Rivera-Batiz, 

1992). Given the negative consequences of failing to complete developmental 
mathematics, it is critical to identify potential ways to improve developmental students’ 
math success. Other working papers in the CCRC Assessment of Evidence Series discuss 
potential ways to improve developmental course completion through improved entry 
assessment (Hughes & Scott-Clayton, 2011), accelerated course structures (Edgecombe, 
201 1), and contextualized curricula (Perin, 201 1). This paper examines the evidence base 
on another potential means for improving learning outcomes and course completion 
among developmental mathematics students: reforms to mathematics classroom 
pedagogy. 
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While national studies that doeument the common features of developmental 
education classroom instruction do not exist (Levin & Calcagno, 2008), typical 
developmental math pedagogy is thought to rely largely on procedural skill-building 
(Goldrick-Rab, 2007; Hammerman & Goldberg, 2003). Observational studies at 
community colleges in California found that mathematics instruction was characterized 
by review, lecture, independent seat- work, and math problems devoid of application to 
the real world (Grubb, 2010; Grubb & Worthen, 1999). Although traditional features of 
math instruction have been linked to better performance on standardized tests and much 
of the mathematics we encounter in our lives requires the ability to use algorithms to 
quickly and accurately solve computations, in order to understand mathematics, students 
need much more than procedural fluency (Hiebert & Grouws, 2007; Kilpatrick, Swafford, 
& Findell, 2001). 

Kilpatrick et al. (2001) identified five interdependent strands of mathematical 
learning that instructional practices must address to build mathematical proficiency: 

1. conceptual understanding — the understanding of why and 
when a mathematical idea is important or useful, 

2. procedural fluency — the ability to use procedures in the right 
way and for the right purpose, 

3. strategic competence — problem formulation and 
representation, 

4. adaptive reasoning — logical reasoning about mathematical 
relationships, and 

5. productive disposition — the belief that a sustained effort in 
learning mathematics will lead to greater understanding and 
benefit one’s life. 

This paper discusses forms of instruction that are thought to support components of 
mathematical learning beyond procedural fluency and should, therefore, develop 
mathematical proficiency more effectively than traditional instruction. 
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1.1 Studies Included in the Review 



Although the purpose of this literature review is to identify promising 
developmental math pedagogy, there is very little empirical research on this topic and 
population specifically; thus, I also reviewed the literature on mathematics pedagogy in 
elementary and secondary schools and in college-level courses, focusing on empirical 
studies that evaluate the impact of an instructional practice on student outcomes.' The 
elementary and secondary school math pedagogy literature is included not only because it 
is more prevalent than empirical research on math instruction in higher education but also 
because recommended best practices in teaching are often similar across grades and even 
subjects. For example, important works that outline best practices for developmental 
education (e.g., Blair, 2006; Boroch et al., 2007; Boylan, 2002) and K-12 mathematics 
teaching (e.g., Donovan & Bransford, 2005) provide similar recommendations: that 
educators connect new knowledge to prior learning; use a variety of instructional 
methods, including learner-centered activities; and provide students with feedback 
through the use of ongoing assessment. Reviewing the K-12 empirical literature 
illustrates how these general recommendations are implemented in the classroom in ways 
that are effective or ineffective for primary and secondary students. These findings can 
then be used to make recommendations regarding how pedagogy that works in the K-12 
math classroom could perhaps be adapted for adult students in the developmental 
education classroom in an effort to apply commonly accepted best practices in teaching. 

It is important to note that articles on curriculum reforms that may also lead to or 
require changes to pedagogy are not included in this review. Curriculum reforms 
typically can involve so many interconnected instructional changes that it is difficult to 
isolate the effects of the individual reforms (Hamilton et al., 2003). For example, this 

* The inclusion criteria for this study encompass qualitative and quantitative (excluding single-subject 
design) studies published from 1990 to the present whose target population includes students in K-12 
schools and institutions of higher education in the United States. To find publications within the topic 
scope, major databases and websites were searched using the term mathematics in combination with 
instruction or pedagogy or in combination with instructional (or pedagogical) practices or strategies. 
Databases and websites included: EBSCO’s Academic Search Premier, Education Research Complete, 
Education Eull Text (Wilson), ERIC, JSTOR, ProQuest, state higher education websites, community 
college system websites, Mathematical Association of America (MAA), American Mathematical 
Association of Two-Year Colleges (AMATYC), and Google Scholar. I also conducted a manual search of 
the following pertinent publications: Journal of Developmental Education, Remedial and Special 
Education, Community College Journal of Research & Practice, Community College Review, and 
Community College Journal. 
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paper does not review a study that evaluated the impact of a reform-based undergraduate 
calculus textbook (Darken, Wynegar, & Kuhn, 2000), which includes exercises that use 
group work and technology and emphasize conceptual understanding. The authors did not 
report the extent to which instructors who volunteered to use these texts incorporated 
these changes to pedagogy in their classrooms, and it is impossible to understand the 
discrete impact of each pedagogical change on the overall results. In contrast, while many 
studies included in this review required changes to content (e.g., using multi-step word 
problems rather than problems that require only a few steps to solve), they focus on the 
methods teachers use to teach the content. 

1.2 Assessing the Quality of Evidence 

In reviewing papers, I critiqued the findings according to the strength and rigor of 

the research designs of the studies in order to evaluate the direction and quality of 

evidence on math pedagogy. Table 1 (found in the appendix) provides a concise summary 

of each reviewed article, including the author’s findings, effect size of the results, and 

methodological problems that weaken the internal validity of the results. This review 

describes the results of rigorous studies and draws conclusions about effective pedagogy. 

In general, to be considered rigorous, a study had to be transparent about the 

comparability of the treatment and control conditions, providing some confidence that 

any outcomes are the result of the instructional intervention, not differences between the 

treatment and control groups. Most importantly, rigorous studies either demonstrated that 

there were no pretreatment ability differences between students who received the 

instructional intervention and students in the comparison group or statistically controlled 

for pretreatment ability, and they made an effort to assign similar instructors to the 

treatment and comparison groups. The results of the remaining studies are inconclusive 

because they could be due to differences between the treatment and control group rather 

-1 

than the instructional intervention. 



^ There were four methodological flaws commonly found across studies: student design non-equivalency, 
teacher design non-equivalency, possible attrition issues, and lack of a comparison group. The notes for 
Table 1 include a detailed explanation of each of the four main flaws. 

^ Except for a few higher education studies, the results of the non-rigorous studies are not described in this 
review, but they can be found in Table 1. 
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1.3 Organizational Framework 

There are several ways to categorize instructional practices. Grubb (2010) 
organizes practices along a behaviorist-constructivist continuum. Commonly accepted 
groupings of instructional practices include broad categories, such as student-centered 
versus teacher-directed instruction or direct instruction versus inquiry-based teaching 
(Hiebert & Grouws, 2007). The current paper uses a more detailed classification system 
driven by specific theories of learning, which provide concrete explanations of particular 
mechanisms underlying different pedagogical approaches. Based on an inductive 
approach, studies were organized into six sets, each containing similar types of practices 
that share a clearly identifiable (although not always explicitly identified by each study’s 
authors) theory of learning. Each set of practices is therefore supported by a particular 
theory that explains why these practices should lead to improved math learning and 
understanding. The six sets are student collaboration, metacognition, problem 
representation, application, understanding student thinking, and computer-based 
learning.^ 

In the first section, for each of the six sets, I explain the relevant theories that 
support the pedagogical practices in the set, provide an example of how the theoretical 
concepts are applied in the developmental math classroom today, and evaluate the 
empirical evidence from the articles in the set. Subsequent sections describe the 
recommendations based on the review of this literature, including methodological 
recommendations and new directions for developmental math pedagogy research. The 
paper concludes with examples of instructional changes to the developmental math 
classroom that may contribute to improved outcomes for developmental math students. 



The practices across the six sets can interact in the classroom and overlap in myriad ways. Hiebert and 
Grouws (2007) emphasize that “teaching is a system of interacting features” (p. 374); different features do 
not fit easily into categories. For this paper, studies were categorized according to the feature by which they 
were most clearly defined. For example, in the application set, instructional practices always involve 
students working together and often promote metacognition and problem representation skills. Studies in 
the application set are distinct from studies in the other five sets because instruction is characterized by the 
use of elaborate problem-solving activities that use real-world contexts. This type of pedagogy is supported 
by distinct theories of learning, suggesting that student learning in classrooms that use real-world problem 
solving is enhanced in different ways than learning in classrooms that use student collaboration, promote 
metacognition, and/or build problem representation but do not use real-world problem solving. 
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2. Theory, Practice, and Empirical Evidence 



2.1 Student Collaboration 

This first set of studies evaluates different models of student collaboration in the 
math classroom. Some studies examine the impact of informal forms of student 
collaboration, while others evaluate the effectiveness of more structured forms of peer 
collaboration, such as Math Excel, Peer-Led Team Learning, the Learning Together 
model of cooperative learning, peer-assisted learning strategies (PALS), peer-mediated 
instruction (PMI), and Team Accelerated Instruction. 

Theory. Springer, Stanne, and Donovan (1999) organize the theories that support 
student collaboration into three theoretical perspectives that identify the different 
mechanisms that link small-group instruction to improved student outcomes. The 
motivational perspective supports the notion that in competitive learning environments, 
the probability of success decreases at an increasing rate as other students succeed, but in 
small-group learning environments, success is dependent on students working together to 
achieve a common goal. The affective or humanist perspective explains that interaction 
among students leads to a nonthreatening environment in which underrepresented 
students have more opportunities to participate and learn. Linally, according to the 
cognitive perspective, student collaboration on open-ended questions leads to greater 
cognitive growth, and the act of explaining material to another student is one method of 
cognitive elaboration, which facilitates the retention of information. 

Johnson and Johnson’s social interdependence theory, which is cited in a number 
of empirical and instructional publications on cooperative learning in higher education 
math courses (e.g., Arendale, 2004; Dees, 1991; Norwood, 1995; Summers & Svinicki, 
2007; Zachry, 2008), draws from all three theoretical perspectives (motivational, 
affective, and cognitive). Negative interdependence, which is characterized by a 
competitive learning environment, can potentially be demotivating because students can 
only succeed if others are failing (Johnson, Johnson, & Smith, 1991). In addition, in both 
classrooms with no interdependence (where students work on their own) and classrooms 
with negative interdependence, students do not benefit from an improvement in social 
skills and cognitive growth that is thought to result from positive social interactions and 
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the exchange of information between peers (Johnson et ah, 1991). On the other hand, 
positive interdependence, where the success of one individual is dependent on the success 
of others, is found in cooperative learning environments characterized by face-to-face 
interaction, personal responsibility in working toward a shared goal, the use of 
interpersonal skills, and group processing through the exchange of feedback, 
explanations, and other information (Johnson et ah, 1991). In structured cooperative 
learning situations with these elements, motivational, affective, and cognitive 
mechanisms are thought to lead to improvements in learning outcomes. 

Student collaboration methods are also supported by Lev Vygotsky’s theory of 
the zone of proximal development and the theory of constructivism. The zone of 
proximal development is the cognitive space between what a learner can do 
independently (i.e., their ability) and what a learner can do with the aid of a teacher or 
through peer collaboration (i.e., their potential) (Safford-Ramus, 2008). Constructivism 
describes learning as a process in which knowledge is constructed through building on 
prior experiences, engaging in self-discovery, and collaborating with peers (Yilmaz, 
2008). Collaborative learning in developmental math education is a way to both expand 
students’ zone of proximal development and apply the principles of constructivist 
learning (Casazza, 1998; Norwood, 1995; Safford-Ramus, 2008). 

Empirical evidence. For the most part, the 15 studies in this set (see Table 1) 
found that student collaboration has a positive impact on math learning. Most of these 
studies did not ensure treatment and control group equivalency; thus, their results could 
be due to factors other than the student collaboration method. However, five rigorous 
elementary school student collaboration studies demonstrate that highly structured forms 
of student collaboration are especially effective for low-achieving math students.^ 

With PALS and PMI, teachers train students to follow a routine that involves 
students taking turns, acting either as the tutor who asks questions and provides feedback 
at every step of the problem-solving process or as the tutee who answers questions at 
every step (Fuchs, Fuchs, Phillips, Hamlett, & Karns, 1995; Fuchs et ah, 1997; Fuchs, 
Fuchs, & Kams, 2001). Lower- achieving students are usually paired with higher- or 

^ The Calhoon and Fuchs (2003) study is also rigorous but evaluates the effects of PALS and curriculum- 
based measurement (CBM) in combination and, therefore, the outcomes cannot be attributed strictly to 
student collaboration. 
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average- achieving students. Across the three studies of PALS and PMI (Fuchs et ah, 

1995; Fuchs et ah, 1997; Fuchs et ah, 2001),^ results varied for students with different 
levels of math aptitude. Students with learning disabilities and average- achieving 
students usually made small gains in achievement compared to their counterparts in the 
control classrooms. Lower- achieving students benefited the most from the different 
student collaboration treatments, making small to moderate gains on all tests but one that 
measured application skills in one study. Higher-achieving students only benefited from 
paired work when more complex math tasks were included in the PMI treatment. 
Similarly, a randomized study by Ginsburg-Block and Fantuzzo (1998) found that low- 
income, low-achieving third- and fourth-grade students who used a highly structured 
peer-tutoring format to review and reinforce math skills outperformed comparable 
students who received traditional instruction. In contrast, Karper and Melnick’s 
randomized study (1993) in a wealthy school district found that the cooperative learning 
technique Team Accelerated Instruction had no impact on the math achievement of 
students in the treatment group. Taken together, the five studies suggest that structured 
student collaboration may be more beneficial for low-achieving elementary school 
students struggling with math. 

Finally, the highest quality developmental education study in this review (Dees, 
1991)^ suggests that an instructor-designed, structured student collaboration method may 
be a promising practice for the developmental math classroom. Dees (1991) randomized 
over 70 students in her developmental math course into four laboratory sections taught by 
graduate assistants: two that used small-group instruction and two that used teacher- 
directed instruction. Students in the cooperative learning lab outperformed students in the 
control group on teacher-made tests and a standardized final exam; however, the internal 
validity of the study’s findings is undermined by the lack of detail about the research 
design, such as how graduate assistants were assigned to sections. 

® One concern with these studies is that although teachers were randomized to the treatment and control 
classrooms and implemented the treatment with the whole class, due to resource constraints only a subset 
of students were chosen to take the posttest. However, these studies show that there are no statistical 
differences along observable characteristics between the treatment and control students in the whole class 
and subset of students. 

^ The Dees article meets the inclusion criteria because it was published after 1990; however, it is important 
to note that the study took place in 1982, so it is much older than most of the studies included in this review 
that were conducted during the past 20 years. 
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student Collaboration in Practice 



Peer-Led Team Learning (PLTL) was first established at the City University of New York and has since been 
used in math and science courses at colleges across the country (Arendale, 2004). Through the PLTL 
program, advanced students are trained in teaching and cooperative learning techniques, becoming "peer 
leaders" who lead weekly collaborative workshops with small groups of students (Arendale, 2004). The 
developmental math faculty at Mountain Empire Community College (MECC) in Virginia have 
implemented PLTL in the developmental math course Algebra L* Peer leaders are assigned to sections of 
Algebra I, which they attend for one hour on Mondays and Wednesdays. Students from these courses 
who have chosen to participate in PLTL meet with their peer leaders during the PLTL session, named 
"Power Hour," held on Fridays directly before their Algebra I course. During the PLTL sessions, the peer 
leaders and students work cooperatively in small groups on areas of Algebra the students are struggling 
with. A PLTL leader who was a former tutee in the program spoke about her experience with PLTL: "I love 
math. I'm a math tutor. But after being out of school for so long, it was like reading Hebrew the first 
semester, and the Power Hour made me feel more comfortable. I could understand mostly what my 
instructor told me, but it was just a different language coming from my PLTL leader." 

*lnformation on PLTL is from a Community College Research Center site visit to MECC in spring 2010. 



2.2 Metacognition 

The second set of studies includes instructional practices that promote 
metacognition, or an awareness of one’s own thought processes, through comprehension 
monitoring, cognitive strategy instruction, or using writing and questioning during the 
problem-solving process to foster self-reflection. 

Theory. The connection between metacognition and math learning is supported 
by a number of theories. Garofalo and Lester (1985) extended Flavell’s (1979) theory 
about the role of metacognition to identify its two main aspects, “knowledge of 
cognition” and “regulation of cognition” (p. 164), and apply them to the mathematical 
problem-solving process. In Garofalo and Lester’s (1985) cognitive-metacognitive 
framework, the knowledge of cognition begins in the orientation phase, in which the 
problem solver assesses the information given about the mathematical task, his or her 
level of familiarity with the task, the difficulty of the task, and possible strategies to use. 
Next, in the organization or planning phase, the regulation of cognition aids the problem 
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solver in connecting his or her understanding of the concept to an understanding of the 
appropriate strategies to use to solve the problem. In the execution stage, the problem 
solver solves the problem using the necessary strategies and procedures. Finally, during 
the verification stage, the problem solver confirms that the solution is correct by checking 
the computations and problem-solving process for errors. Garofalo and Lester (1985) 
argue that mathematical problem solving can be improved by training students to 
incorporate the stages of metacognition into their problem-solving process. 

The theory of information processing provides another justification for the 
importance of metacognition, contending that learning occurs as individuals think about 
their own thinking as they retrieve stored information from memory (i.e., prior 
knowledge) and use it to process new information (Safford-Ramus, 2008). Metacognition 
has also been integrated into forms of constructivism. For example, in Narode’s (1989) 
description of instruction in a constructivist developmental math program for students at 
the University of Massachusetts, Amherst, he writes, “the method of instruction 
incorporates two key notions: constructivism, the idea that students must construct 
knowledge for themselves, and metacognition, the supposition that the vehicle for the 
construction of knowledge is self-reflection” (p. 6). In the math classroom, instruction 
that uses the cognitive-metacognitive framework would emphasize not only each 
student’s ability to solve problems but also each student’s capacity to assess a problem’s 
difficulty, choose the appropriate strategy or strategies to solve a problem, engage in self- 
monitoring during the problem-solving process, and evaluate the final solution for its 
accuracy. 

Empirical evidence. The eight reviewed studies (see Table 1) describe unique 
pedagogical practices that seek to improve students’ ability to monitor their problem- 
solving process, but only one study employed a rigorous experimental design. In 
Tournaki’s (2003) study, second-grade students with and without disabilities were 
randomized into a control group and two treatment groups in which, in addition to their 
regular class time, students received either explicit instruction in either strategy 
instruction (verbalizing problem-solving steps), or drill-and-practice strategies. Students 
with learning disabilities who received strategy instruction experienced large gains in 
addition facts performance compared to students with learning disabilities who received 
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drill-and-practice instruction, but there were no differences in addition facts achievement 
between students without disabilities who received the strategy instruction versus drill- 
and-practice instruction. Tournaki concludes that explicit instruction in problem-solving 
strategies, even for tasks as simple as adding, may be especially important for students 
with learning disabilities. 



Metacognition in Practice 

The College Transition Initiative (CTI) at the CUNY community colleges prepares GED graduates to take the 
COMPASS placement math exam, with the related aims of reducing their need for remediation, deepening 
their understanding of algebra, and preparing them for their first college-level math course (Hinds, 2009). 
In addition to a supportive structure in which the CTI math course is part of a learning community and an 
innovative math curriculum, CTI math instruction employs a number of non-traditional pedagogical 
practices. Procedural rules are not taught; in their place, students build their conceptual understanding by 
discussing mathematical relationships, and rules that emerge through these activities are discussed at the 
end of lessons so that students can also build their procedural fluency. The instructor does not lecture; 
instead, the instructor asks students higher order thinking questions about contextualized functions that 
help students transition to more abstract work. For most of class time, students work in groups to solve 
problems and are constantly involved in discussions sparked by metacognitive questioning. Common 
questions asked by the instructor include: 

What did you do? 

Why did you do that? 

Do you agree with what she/he just said? Why? 

Did any of you do it differentiy? Why? 

What do you see? 

Does this remind you of anything? 



2.3 Problem Representation 

The third set of articles encompasses pedagogy that improves students’ problem 
representation skills, such as the use of multiple representations and strategies during the 
problem-solving process, the learning/teaching approach, concept-based instruction, the 
concrete-to-representational-to-abstract (CRA) instructional sequence, schema-based 
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instruction, and the Quantitative Understanding: Amplifying Student Achievement and 
Reasoning (QUASAR) project.^ 

Theory. The cognitive phases of problem solving are analogous to the 
metacognitive framework except that in the first phase, called the problem representation 
phase, the focus is on building a mental representation of the problem rather than 
assessing the problem’s difficulty, strategies to use, and other aspects of the problem- 
solving process (Brenner et ah, 1997). The importance of problem representation is based 
on the cognitive theory that problem solvers must understand the connection between the 
problem and its different representations before they can move on to planning and 
executing the procedural steps to solve a problem (Brenner et ah, 1997; Chappell, 2006). 
By focusing on the procedures needed to execute or solve problems, teachers and 
instructors neglect this critical first phase of problem solving. Higher levels of math, such 
as algebra, require an understanding of how algebraic (or symbolic) representations can 
be represented by graphs and other forms, so students who did not learn problem 
representation skills in lower-level math courses, such as pre-algebra, may experience 
increasing difficulties as they progress in math (Brenner et ah, 1997; Zawaiza & Gerber, 
1993). 

The cognitive phases of problem solving suggest that problem representation 
skills will lead to more success in the use of procedures during the problem execution 
phase, and Rittle- Johnson, Siegler, and Alibali (2001) find empirical evidence suggesting 
that, in fact, problem representation is the mechanism underlying the process by which 
conceptual knowledge can lead to improved procedural fluency, which in turn can lead to 
improvements in conceptual knowledge. In the Rittle- Johnson et al. study, fifth-grade 
students were randomly assigned to receive assignments on placing decimals on a 
number line that provided different levels of representational supports (e.g., a zero-to-one 
number line with no markings versus a zero-to-one number line with the tenths place 
marked). The level of representational supports determined students’ problem 
representation skills, measured by their ability to explain why they chose a spot on the 
number line to represent each decimal, on an intervention test, and their procedural 

* The QUASAR project includes many more instructional components than strategies that improve 
students’ problem representation skills (Silver & Stein, 1996), but the QUASAR articles in this review 
focus on instructional tasks that use multiple representations or solution strategies. 



12 




fluency, measured by marking the position of a decimal on a number line, on a posttest. 
Procedural skill performance was then a significant predictor of conceptual knowledge, 
measured by student performance on new tasks that required an understanding of decimal 
fractions, on a posttest. 

The Rittle- Johnson et al. study (2001) does not suggest a specific instructional 
strategy, such as providing students with worksheets that include representational 
supports. Rather, it supports the theory that students who develop an understanding of 
how concepts can be represented in different, connected ways will be able to link these 
representations to the procedures that are necessary to solve a problem. This will lead to 
more accurate procedural fluency (or solution execution), and solving problems with 
understanding will enhance conceptual knowledge. 

Empirical evidence. Five of the nine studies in this set (see Table 1) are among 
the strongest studies in this review and provide compelling evidence that improving 
students’ problem representation skills has a small to moderate positive effect on math 
learning. The four high-quality studies that took place in elementary and middle school 
classrooms (Brenner et al., 1997; Jitendra et al., 1998; Jitendra et ah, 2009; Witzel, 
Mercer, & Miller, 2003) found support for the routine use of multiple representations 
during problem solving by teachers and students. 

In the only higher education study in this set, Chappell (2006) employed a number 
of methods to ensure that even though students self-selected into concept-based calculus 
and traditional calculus sections (unaware of the instructor or instructional method of 
each section), the faculty and students across both groups were comparable. In addition, 
frequent, unannounced classroom observations by faculty not directly involved in the 
study confirmed that in the concept-based sections, faculty taught students how to solve 
problems using numerical, graphical, and algebraic methods while constantly connecting 
new ideas to prior knowledge. In the control sections, faculty moved through the 
textbook teaching definitions and formulas in a linear manner. Students in the concept- 
based sections performed significantly better on the midterm and final and were better 
able to transfer their understanding to unfamiliar concepts. For example, on a final exam 
problem that had never been introduced in any of the classes, 88% of the students in the 
concept-based classrooms answered this question correctly, and only 3% did not support 
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their answer with an explanation. Only 54% of the students in the traditional sections 
answered this question correctly, with most of them providing textbook definitions to 
explain their answer and 31% not providing any explanation at all. 



Problem Representation in Practice 

The traditional approach to introducing radical equations teaches the procedural skills related to solving 
them, as in the following exercise*: 

Solve lyjx + 2 = 10 

First, isolate the radical by dividing both sides of the equation by 2: 

Vx + 2 =5 

Next, square both sides of the equation: 

{jx + lY = 5 ^ 

x+2 = 25 
x= 23 

Now check for extraneous solutions. 

The following exercise is thought to better promote conceptual understanding of radical equations: 

X f{x) = ylx 

0 vo = o 

1 VI = 1 

2 V2 =1.414 

3 V3 = 1.732 

4 V4 = 2 

5 V5 = 2.236 

7 V7 = 2.646 

9 V9 = 3 

Graph the basic square root function using the data from the input/output table. 

Drawing the graph by hand helps students remember concepts about domain. 

Now draw the graphs of/(x) = l,/(x) = 2,/(x) = 3, and/(x) = -1 all on the same axes. 

This allows students to see the points of intersections (the solutions) and where there are no 
points of intersection (the extraneous solutions). Then, they will be more prepared to solve the 
radical equation below. 

Solve zVx + 2 = 10. 
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Problem Representation in Practice, Continued 

Students should continue to connect procedures and solutions to the graph of the equation by answering 
the following questions: 

What is the basic function? 

What is the transiation or the transformation of the function ? 

Solve f(x) by labeling all Important points on a graph. 

*This example is adapted from Gowribalan A. Vamadeva's presentation "Fostering Conceptual Understanding in a 
Developmental Algebra Classroom" at the 2009 AMATYC Conference proceedings and included with her permission. 



2.4 Application 

Application-oriented instructional approaches include project-based learning, the 
modeling-based approach, the functional approach, enhanced anchored instruction, the 
algorithmic instructional technique, and culturally responsive pedagogy, all of which are 
designed to teach math concepts and skills through real-world problem-solving. Many of 
these instructional approaches may be referred to as “problem-based learning” (Boroch et 
al., 2007, p. 45), which is one form of contextualization (Perin, 201 1). 

Theory. The link between application of math concepts to students’ everyday 
lives and improvements in math performance is supported by the theory of situated 
cognition (The Cognition and Technology Group at Vanderbilt, 1990). Situated cognition 
is based on the idea that since an important objective of schooling is the transfer of .skills 
from the abstract to the concrete, skills and concepts should not be taught without 
reference to the real world; rather, they should be situated in authentic activities (Brown, 
Collins, & Duguid, 1989). Authentic activities value the experiences and knowledge 
students bring with them to the classroom and allow students to learn math in a context 
that is meaningful to them. For example, culturally responsive pedagogy, which utili z es 
authentic, culturally based activities, is believed to be more effective than traditional 
instruction because “it filters curriculum content and teaching strategies through their 
cultural frames of reference to make the content more personally meaningful and easier 
to master” (Gay, 2000, p. 24). 
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The cognitive apprenticeship model of learning explains how pedagogy and 
content (e.g., authentic activities) based on the theory of situated cognition are connected 
to student learning and transfer of skills. In this model, students are apprentices who 
through guided practice become independent learners, and the skills they learn are both 
physical and cognitive, since “knowing and doing” (Brown et ah, 1989, p. 39) are 
inseparable. First, teachers introduce a new concept by modeling how to solve a problem 
that students are fa mi liar with; then, as students become more comfortable with the 
concept, they are given authentic activities to work on in collaborative groups (Brown et 
ah, 1989). Finally, as opposed to traditional instruction that begins with equation solving 
and ends with word problems that situate math in real-world contexts, instructional 
approaches based on situated cognition may end with students independently using 
algorithmic procedures to solve problems that assess the understanding of the same math 
ideas and skills embedded in the authentic activities (Brown, et ah, 1989; Laughbaum, 
2003; Vasquez, 2003). In other words, this learning process is thought to aid in the 
transfer of math skills to both procedural problem solving and open-ended, realistic 
problem-solving activities (Boaler, 1998). 

Another mechanism that may explain why application-oriented instructional 
approaches in this set may improve math learning is motivation. Motivation may be the 
result of pedagogical techniques, such as collaborative learning (Johnson et ah, 1991; 
Springer et ah, 1999), and the contextualized curriculum (Perin, 201 1). The practices in 
this set may improve student learning through different mechanisms because they are 
defined by a number of instructional and curricular supports, including instructor 
scaffolding, complex problem-solving embedded in real-world or culturally based 
situations, collaborative group work, and hands-on activities that use technology that 
helps students contextualize math concepts in real-world problems (Bottge, Heinrichs, 
Chan, & Serlin, 2001; Bottge, Heinrichs, Mehta, & Hung, 2002; Brenner, 1998; 
Ellington, 2005a, 2005b; Ganter & Jiroutek, 2000; Hickey, Moore, & Pelligrino, 2001; 
Hollar & Norwood, 1999; Kennedy, Vasquez, & Huber, 2003; Laughbaum, 2003; Lipka 
& Adams, 2004; O’Callaghan, 1998; Shore, Shore, & Boggs, 2004; Vasquez, 2003, 
2004). 
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Application in Practice 



At Maricopa community colleges, students in arithmetic review courses were given two weeks to solve 
the following problem in groups (Tannehill & Zeka, 1997): 

You are interested in purchasing a new vehicie. 

What shouid your annuai salary be to afford the car you want? 

Instructors acted as facilitators, providing students with web resources and formulas to help them 
calculate their dept-to-income ratio, car costs, loan payments, and other critical pieces of information that 
students needed to be able to calculate how much they would have to earn to finance the car they want. 



Empirical evidence. The eleven studies in this set (see Table 1) consistently find 
a positive association, with trivial to large effect sizes, between teaching math through 
application and improved performance on tests of conceptual understanding. However, 
only the Hickey et al. (2001) study attempted to ensure that the treatment and control 
conditions and students and teachers in both groups were comparable, and the treatment 
effects were too small for the results to conclusively support the use of a video series that 
takes elementary students on complex math adventures. In many of the studies in this set, 
treatment students outperformed control students on tests of understanding, but there 
were no differences in performance on more traditional tests of procedural fluency 
(Bottge et ah, 2001; Bottge et ah, 2002; Hickey et al., 2001; Hollar & Norwood, 1999; 
O’Callaghan, 1998) — a finding that is consistent with studies in the problem 
representation set (Brenner et al., 1997; Chappell; 2006). These studies highlight the 
possible trade-offs that are made when reform-based instructional practices are used in 
the math classroom at both the K-12 and college levels. While these practices show 
promise for improving students’ conceptual understanding, which is considered a 
necessary condition to be successful in math at any level (Bransford, Brown, & Cocking, 
1999; Hiebert & Grouws, 2007; Katz, 2007; Kilpatrick et al., 2001), they may require 
more time and preparation and sometimes do not improve students’ computational .skills 
(Brenner et al., 1997). 
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2.5 Understanding Student Thinking 

The fifth set of practices encompasses instructional methods that help teachers 
understand student thinking and adjust their instruction to meet the needs of their 
students. These pedagogical practices include assessment methods that are used during 
instruction to monitor student progress and guide instruction, such as frequent testing, 
classroom assessment techniques, classroom voting, the Keystone Method, progress 
monitoring, and curriculum-based measurement (CBM). 

Theory. While cognitive and behavioral theories explain the relationship between 
assessment and students ’ understanding of their own thinking, theories and applications 
of cognitively guided instruction (CGI) explain the importance of teachers ’ 
understanding of student thinking (Bransford, Brown, & Cocking, 1999; Carpenter, 
Fennema, Peterson, Chiang, & Loef, 1989; Villasenor & Kepner, 1993). CGI is the use of 
instructional practices that are informed by research on how students solve math 
problems and by knowledge of the prior math skills and misconceptions students bring to 
the math classroom (Carpenter et ah, 1989; Villasenor & Kepner, 1993). CGI is based on 
a cognitive perspective of teaching that examines the interaction between teachers’ 
instructional choices, their knowledge of student thinking, and student performance 
(Carpenter et ah, 1989). The cognitive view of teaching hypothesizes that teachers who 
have an understanding of their students’ thinking (as well as strong subject matter and 
pedagogical content knowledge) will make better instructional choices that result in 
improvements in student math achievement (Ball & Bass, 2000; Bransford et ah, 1999; 
Carpenter et ah, 1989; Kieran, 2007). 

Applying the cognitive view of teaching can be challenging, however, since 
monitoring the progress of a classroom of students “create[s] an overwhelming demand 
on the cognitive resources of the teacher” (Carpenter et ah, 1989, p. 501). As a result, 
teachers are only able to make small adjustments in their instruction based on their 
assessment of student understanding and thinking. However, today, formative 
assessment, or the adjustment of instruction based on performance of students, can be 
driven by technological tools that provide an efficient means of monitoring student 
progress through frequent assessment, especially at the college level (Blair, 2006; Boroch 
et ah, 2007; Cline, 2006). Technological supports, such as clickers (see, e.g., Cline, 
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2006), are aligned with a cognitively based perspective on instruction that hypothesizes 
that teachers make more effective instructional choices that help students build their 
mathematical understanding when they are able to assess students’ prior knowledge and 
current understanding of the material. 

Empirical evidence. While it is generally accepted that college instructors should 
adapt instruction to meet the needs of their students through meaningful, ongoing 
assessment (Adams, 1997; Blair, 2006; Boroch et al., 2007; Boylan, 2002; Siadat et al., 
2008), the four studies targeting college and developmental education students did not 
utilize comparison groups or compared non-equivalent groups (see Table 1), so their 
findings cannot confirm the positive impact of ongoing, formative assessment. Among 
the nine K-12 studies (see Table 1), the rigorous studies by Fuchs and Fuchs (1990), 
Fuchs, Fuchs, Hamlett, and Stecker (1991), and Fuchs, Fuchs, Hamlett, Phillips, and 
Bentz (1994) find strong support for using CBM that provides teachers with expert 
recommendations to make instructional changes with elementary school students with 
learning disabilities. However, given that all three studies were conducted by the same 
researchers in the same setting, confidence in the validity of these studies would be 
strengthened if their results were replicated elsewhere. 



Understanding Student Thinking in Practice 

Daley College implemented the Keystone Method in Elementary Algebra, Intermediate Algebra, and 
College Algebra (Siadat et al., 2008). This instructional method utilizes daily assessment of students to 
inform the instructor of the progress of the class and each student. The instructor can then adjust 
instruction accordingly and inform students of their individual progress. Before each class, the instructor 
conducts an item analysis of student answers on the quiz questions that tells the instructor which areas 
should be retaught or reviewed the next day and which problems to include on future quizzes. If the 
standard deviation of the quiz scores is greater than 0.25 (i.e., if there is considerable divergence in 
student understanding of specific skills), the instructor creates small, heterogeneous groups of students 
with one from each of the quiz performance quartiles. As a result, groups have weak, average, and high- 
performing students who can motivate and help each other learn the material. If quiz scores reflect a 
general understanding of skills across all students, the instructor uses more traditional modes of teaching 
to move on to new concepts. 
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2.6 Computer-Based Learning 

This last set includes studies in which students work through technology- 
delivered mathematics content at their own pace during some or all of classroom time, 
with the instructor providing some face-to-face interaction through individualized 
attention, delivery of instruction, or technology support. Computer-based learning may 
also encompass many of the approaches previously discussed, such as using computer- 
based technology to provide students with real-world problem-solving opportunities and 
monitoring the progress of students in order to guide instruction and content. Computer- 
based learning includes course redesign models, a popular trend in developmental 
mathematics where some or all of face-to-face instruction is replaced with a set of self- 
paced, online curriculum modules (Epper & Baker, 2009; Twigg, 2005); hybrid or 
blended online learning; and forms of computer-based learning where the traditional 
course structure is maintained and the instructor still has a role in the classroom. 

Theory. Originally, computer-based instruction was based on the theory of 
behaviorism (Hung, 2001; Safford-Ramus, 2008). According to behaviorism, responses 
to stimuli (e.g., questions) that are directly followed by positive or negative 
reinforcement will lead to the conditioning of consistent, correct responses representing 
the learning of material (Safford-Ramus, 2008). In one of the first educational 
applications of behaviorism, a “teaching machine” provided students with academic 
material followed by factual questions; students’ responses were then fed back into the 
machine (Skinner, 1960). The machine provided immediate responses to each answer: the 
student received new material for correct answers and the same question for incorrect 
answers. Skinner (1960) believed that this type of programmed instruction motivated 
learning by breaking down concepts into small, manageable pieces of information that 
students could work through at their own pace while continually receiving immediate 
feedback on their understanding. 

Computer-based learning has evolved since the advent of Skinner’s teaching 
machine. First, there is a range of pedagogy inherent to instructional software programs. 
Some computer-based tutorial and learning programs deliver drill- and-practice exercises 
(Hung, 2001), while other instructional software programs provide problem-solving 
activities that emphasize deeper understanding of mathematical concepts (Epper & 
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Baker, 2009; Hung, 2001; Stillson & Alsup, 2003; Twigg, 2005). As a result, technology- 
mediated instructional content may determine the extent to which students experience a 
balanced learning environment where procedural skills, conceptual understanding, and 
other components of mathematical proficiency are all addressed. Second, computer-based 
learning can be designed to incorporate principles of constructivism (Kanuka & 
Anderson, 1999). For example, because of its self-paced learning component, computer- 
based instruction is called a student-centered model of learning (Trenholm, 2006; Zhu & 
Polianskaia, 2007). Computer-based learning utilizes an important constructivist 
principle — that students are active in the construction of knowledge rather than passive 
recipients of knowledge, and educators serve as their guides and helpers (Kanuka & 
Anderson, 1999). Other essential features of a constructivist classroom are discovery- 
based learning and meaningful interactions between students, which can also be 
incorporated into computer-based instructional models (Kanuka & Anderson, 1999). 
Finally, the Open Learning Initiative (OLI), which creates online courses for students at 
Carnegie Mellon University, has demonstrated how computer-based delivery of course 
content can incorporate instructional elements that traditional instruction cannot. For 
example, the structure of the OLI-Statistics course is influenced by the theory that 
instructional design should try to eliminate “extraneous cognitive load” (Lovett, Meyer, 

& Thille, 2008, p. 6), or tasks and information that are unnecessary to learn a concept or 
skill. Therefore, the course explains statistics concepts through animations that are 
accompanied by verbal explanations so that students do not have to process separate 
visual and text-based explanations of statistics concepts (Lovett et ah, 2008). 

Computer-based learning is also a form of mastery learning (Hagerty & Smith, 
2005; Trenholm, 2006). Under the mastery learning approach, course content is divided 
into small units, and students must demonstrate mastery of one unit before they can move 
on to the next unit (Kulik, Kulik, & Bangert-Drowns, 1990). The mastery learning 
approach is thought to be more effective than traditional instruction because it is tailored 
to each student’s needs; students only work on material they are ready to learn (Kulik et 
ah, 1990). Computer-based instructional programs allow instructors to efficiently provide 
their students with a mastery learning experience through individualized learning 
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programs that monitor student progress and adjust the content delivered accordingly 
(Hagerty & Smith, 2005). 

Empirical evidence. The evidence regarding the impact of computer-based 
learning is inconclusive because none of the 13 studies in this set (see Table 1) employed 
a rigorous research design. Lovett et al. (2008) randomized students in their study on the 
impact of the OLI-Statistics course, but because the treatment condition is an eight- week 
OLI-Statistics course and the comparison condition is a 15- week traditional course, it is 
impossible to disentangle the effects of the OLI pedagogy from the possible effects of 
acceleration. In other words, the eight-week structure of the course may have contributed 
to the outcomes either by motivating students or through another mechanism. The highest 
quality study used a quasi-experimental design and found that college algebra students in 
course sections using ALEKS (Assessment and Learning in Knowledge Spaces), a 
computer- adaptive, online assessment and learning program, experienced small gains in 
math learning compared to students in traditional algebra course sections (Hagerty & 
Smith, 2005). However, the final results only include students who took both the pretest 
and the posttest, not accounting for the test scores of students who enrolled late or 
withdrew from the course. As a result, differential attrition could have biased the results. 



Computer-Based Learning in Practice 

Cleveland State Community College redesigned its developmental and college math courses by replacing 
three hours per week of class time with one hour per week in a computer classroom with faculty and two 
hours per week in a large computer lab (Squires, Faulkner, & Hite, 2009). Students use MyMathLab to 
work through 10-12 curriculum modules on their own by watching an instructional video, completing 
homework, and then passing a quiz. Based on the mastery learning approach, to move to the next 
module, students must earn at least a 70% on the homework and quiz for each module. Faculty teach 10 
math sections each and spend 10 hours per week in the computer lab providing students with one-on-one 
attention and individualized instruction. The Community College Futures Assembly awarded Cleveland 
State the Bellwether Award in the Instructional Programs and Services category for its redesigned math 
courses. 
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3. Research Recommendations 



Evidence from this literature review supports the National Mathematics Advisory 
Panel’s (2008) conclusion that there is a crucial need for more methodologically rigorous 
scientific research in the area of effective math instructional practices. All of the 
pedagogical practices discussed in this review may have the potential to improve the 
outcomes of developmental math students, but given the poor internal validity of many of 
the studies, it is difficult to infer whether most of the pedagogy is effective in practice. 
The following are four methodological recommendations that could improve the internal 
validity of future research on developmental math instruction. The principal issues and 
patterns of findings that emerged from this review also highlight three directions for 
research on developmental math pedagogy. 

3.1 Methodological Recommendations 

Since it is difficult to conduct randomized experiments in K-12 and college 
settings, researchers should collect information on student abilities and demographics and 
control for any observable differences between groups using statistical methods. Doing so 
would provide more convincing evidence that differences in outcomes are due to the 
instructional intervention rather than preexisting differences between treatment and 
control students. Second, it is recommended that, if the treatment takes place across 
multiple classrooms, participating instructors teach both a treatment section and a control 
section, and that instructors be similar along observable measures of teacher quality. This 
would help ensure that the effects of an instructional practice can be more credibly 
disentangled from the impact of individual instructors. 

Third, multiple-choice math tests and other standardized assessments are often 
used to study the effects of reform-based math instructional practices, but traditional 
assessments may neglect to measure the types of skills that reform-based pedagogy 
promotes (Hamilton et ah, 2003). However, assessments that are designed specifically for 
a study to measure the impact of a single pedagogical practice may not be fair outcome 
measures if they only emphasize the skills and knowledge that were taught in the 
treatment classrooms. It may be most appropriate, therefore, to use multiple outcome 
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measures that include standardized and thoughtfully designed alternative measures of 
math achievement and learning.^ 

Finally, in higher education research, it is important to consider that even if 
students are unaware of which course sections have been assigned to the treatment or 
control group, course scheduling can influence the types of students that register for each 
course, and the characteristics on which these students differ may be related to 
educational performance. Therefore, offering treatment and control sections at similar 
times reduces the likelihood that any outcomes are partly determined by student 
characteristics related to the time of day the course sections are offered. 

3.2 Directions for Future Research 

Conducting rigorous evaluations of computer-based learning in developmental 
math is essential to furthering our understanding of how this popular developmental 
mathematics reform affects student math learning, persistence, and other outcomes. 
Anecdotally, some community colleges have experienced improved pass rates and 
persistence for developmental math students after the introduction of computer-based 
instruction and course redesign (see, e.g., Speckler, 2008; Squires et ah, 2009; Twigg, 
2005). But many questions remain regarding exactly how these new models of 
developmental education are connected to observed outcomes. Course redesign models 
involve substantial changes to the way course content is delivered by replacing some or 
all of the traditional course structure with self-paced online learning modules (Epper & 
Baker, 2009; Twigg, 2003). As a result, outcomes may be due to any number of changes 
in how course content is delivered, when students can access course content, and the 
pedagogy utilized in each model. For example, in some models, students are able to 
access course content at any time from home or in a large computer lab, while in others, 
they must work through the content during structured lab times (Twigg, 2003, 2005). 
Regarding pedagogy, there is variation in how much time students spend working 

® The midterm and final exams in Chappell’s (2006) study are examples of well-designed outcome 
measures. A faculty member not involved in the study designed the midterm and final that tested 
knowledge and skills that were covered in both the control and treatment classrooms, and six other faculty 
members not involved in the study assigned each item to the procedural skill subscale or conceptual 
understanding subscale. Then, the four instructors in the study designed a rubric and graded the exams 
together, ensuring that the students in the treatment and control sections were assessed fairly. 
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through the course material on their own, and some models emphasize individualized 
attention from instructors or tutors and small-group work more than others (Twigg, 2003, 
2005). Differences in pedagogy also extend to the type of instructional software used 
(Hung, 2001): some computer-based course content may allow for investigative problem 
solving or discovery-based learning, while other software programs rely on drill-and- 
practice problems.^*’ Finally, all of these components of course redesign may impact 
students in different ways, and they may even have differential effects for different types 
of students. Future research should aim to isolate the effects of different components of 
course redesigns or to assess the possible differential effects of this reform on student 
subgroups. 

Next, unlike the studies in the application and computer-based learning sets, 
which typically take place in colleges and universities, the studies in the other sets target 
diverse student populations, including elementary, secondary, developmental education, 
and college students as well as students with learning disabilities at each of these 
educational levels. Across all these sets, there are slightly larger effect sizes from high- 
quality studies whose target population is students with learning disabilities compared to 
high-quality studies whose target population is typical students, and within rigorous 
studies that compare outcomes between subgroups of students, effect sizes are slightly 
larger for lower-achieving students or students with learning disabilities. It is important to 
note that many of studies focusing on students with learning disabilities are by the same 
set of authors (i.e., Fuchs et ah). Nonetheless, this pattern may be indicative of the 
importance of pedagogy that utilizes cooperative learning, teaches students to monitor 
their understanding as they solve problems, improves problem representation, and 
involves ongoing assessment and the adjustment of instruction for students struggling 
with mathematics. Therefore, a direction for qualitative and quantitative research could 



For example, in a randomized study, Campuzano, Dynarski, Agodini, and Rail (2009) compared the 
effects of different software products (PLATO, Larson, and Cognitive Tutor) on middle school math 
achievement. PLATO is described as relatively behaviorist because it includes only independent practice 
on procedural skill building, while the Larson products address both skill building and problem solving, 
and Cognitive Tutor requires students to use graphs to represent and solve problem scenarios. None of the 
products had significant effects on student math achievement, and standard errors were large, such that one 
cannot say with confidence that their effects were negative, zero, or positive. However, this study still 
presents a model for research that should be performed at the higher education level. 
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be a more focused examination of how these types of pedagogy could be effective in 
building the foundational skills that adult students need to move beyond basic math and 
into more advanced math subjects. 

A final priority for developmental education research is designing and 
investigating the impact of more balanced instructional approaches that promote all 
strands of mathematical learning. This is especially important since the Mathematical 
Association of America (MAA) and American Mathematical Association of Two-Year 
Colleges (AMATYC) recommend replacing traditional college algebra courses with 
modeling-based college algebra courses, in which students solve problems situated in 
real-world contexts by creating and interpreting mathematical models (Katz, 2007). 
However, while the studies in the application set do not suggest anything conclusive 
about the effects of this type of instruction, they do consistently suggest that application- 
oriented instructional approaches may support some strands of mathematical proficiency 
but do not improve procedural fluency. A challenge for researchers and practitioners is to 
develop modeling-based approaches that improve students’ math understanding as well 
as their performance on traditional standardized tests of mathematics achievement. 



4. Instructional Recommendations 

There are a number of studies in the student collaboration and problem 
representation sets that employed rigorous designs with positive results, and, therefore, 
adaptation and evaluation of these pedagogical practices ought to be considered for the 
developmental mathematics classroom. 

4.1 Structured Student Collaboration 

Structured peer-collaboration methods for developmental and college math 
students are already taking place outside the classroom with programs like Peer-Led 
Team Learning (Hooker, 2010) and Math Excel, workshops where students work 
together collaboratively on math problems that reinforce or review material that is 
covered in their regular course (Dick, 2003). However, it is unknown how prevalent 
highly structured peer-collaboration methods are in the developmental math classroom. 
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Many instructors may use cooperative learning in informal ways, but theory and research 
suggest that cooperative learning may not be effective unless it is formally and 
systematically integrated into a course. According to social interdependence theory, 
positive educational outcomes are the result of engagement in frequent, meaningful 
interactions with others for the purpose of working toward a common goal (Johnson et 
ah, 1991), and the rigorous student collaboration studies found that students benefited 
from cooperative learning methods in which all students played a role in working toward 
a shared goal. 

Applications of structured student collaboration in developmental math include 
collaborative problem-solving activities that have a group grade tied to them. Part of the 
final grade in a course may even include group performance on collaborative activities. A 
specific example of a more formal student collaboration activity comes from Dees’s 
(1991) study. In the developmental math lab sections, groups of four to six students 
received only parts of the instructions to a problem, and then students shared with their 
group the information they received. The group had to work together to understand the 
problem instructions and then solve the problem. At the end of the activity, one group 
member was randomly chosen to explain the group’s solution, and the group’s grade was 
based on this explanation, so group members had to collaborate to ensure everyone in the 
group understood the solution steps and final answer. 

Activities like the one from Dees’s classroom — those that ensure that each student 
has a role in accomplishing a task with a group grade — could be used to supplement more 
traditional instructional practices in the developmental math classroom and used in 
combination with other alternative pedagogical practices. For instance, the use of 
ongoing assessment can help determine the specific areas where students need more 
practice, so instead of conducting a whole-class review of the material, an instructor 
could group students together to work on only the specific areas with which they are 
struggling. Requiring students to explain out loud how they arrived at their solution may 
help them start thinking about their own mathematical thinking, thereby incorporating a 
metacognitive framework into their problem-solving process. Exercises could also 
require students to represent the problem situation in several ways in order to develop 
their problem representation skills. 
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4.2 Improving Problem Representation 

There were no developmental education studies that focused on problem 
representation, but rigorous studies involving other student populations demonstrate how 
instructors can improve student learning by integrating problem representation instruction 
into their lectures, interactive board work, and other traditional modes of instruction. For 
example, in a study by Brenner et al. (1997), pre-algebra students were taught and asked 
to represent problems using graphs, diagrams, tables, pictures, and equations and to solve 
problems using multiple representations. Similarly, in a study by Chappell (2006), faculty 
routinely represented calculus concepts numerically, algebraically, and graphically and 
solved them using these representations in their lectures, and students were expected to 
do the same on homework and assessments. Evidence from both of these studies suggests 
that improving students’ problem representation skills is a promising teaching strategy 
for improving math learning outcomes. 

Developmental math instructors may want to consider modeling problem 
situations numerically, algebraically, and graphically in their lessons and expecting their 
students to represent and solve problems in multiple ways on homework and assessments. 
This would require more time devoted to lesson preparation, a change in the content of 
homework and assessments, and lessons that spend more time on each concept, which 
may reduce the time spent working on procedural fluency through solving equations 
(Brenner et al., 1997). It would be useful for researchers to design and evaluate balanced 
instructional approaches that are able to effectively teach all strands of mathematical 
proficiency, but individual instructors can also experiment with how to find ways to focus 
on problem representation while still providing students with practice on traditional 
equation solving. 



5. Summary of Recommendations 

To summarize, identifying effective pedagogical practices in developmental math 
could potentially lead to improved student learning outcomes and, ultimately, improved 
rates of course completion and persistence. However, more rigorous research in the area 
of developmental math education is needed in order to confirm that certain practices that 
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seem promising are indeed effective in the classroom. First, evaluations of course 
redesign in developmental mathematics should attempt to separate the effects of the 
structural and instructional components of the various models of computer-based 
learning, examine how different types of students respond to reform, and consider the 
possible differential impact of instructional software that emphasizes procedural fluency 
and software that emphasizes mathematical understanding and application. Future 
research should also explore how cooperative learning, ongoing formative assessment, 
and strategies that encourage self-reflection and problem representation skills help 
developmental education students build foundational math skills. Finally, research should 
develop and test balanced instructional methods that have an impact on all strands of 
mathematical proficiency. 

In the meantime, the literature yields support for a few recommendations that 
instructors may find immediately useful. For example, instructors should consider using 
structured collaborative problem-solving activities in which each group member has a 
role in working toward a group product or answer. Second, instructors should consider 
representing the same problem numerically, graphically, and algebraically in their lessons 
on a routine basis. Students can then be expected to represent problems in multiple ways 
on in-class exercises, homework, and assessments. 

Currently, the academic outlook for students who enroll in developmental math 
courses is generally unfavorable. Improving outcomes for developmental math students 
will require the continued efforts of researchers and practitioners. The payoff for those 
efforts may be significant, since bringing more effective pedagogy to the developmental 
math classroom could have profound effects on academic outcomes and job attainment 
for developmental math students. 
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School 


Developmental 

Education 


College 

Level 


Student Design 
Non-Equivalent 


Teacher Design 
Non-Equivalent 


Possible 

Attrition 

Issues 


No 

Comparison 

Group 


Student Collaboration 


Baxter, 
Woodward, 
& Olson 
(2001) 

Qualitative 


X 








Whole-class discussions were difficult for 
the low-achieving students to follow, but 
these students were more engaged during 
pair work, especially when they worked 
with an average or high-ability peer. 
However, close observation of pair work 
interactions revealed low-achieving 
students primarily copying their partner's 
work or managing the materials. 


N/A 


N/A 








X 


Beirne- 
Smith (1991) 


X 








Students with disabilities who were 
randomly assigned to work with peer 
tutors using two different instructional 
methods performed better on an addition 
facts assessment than the control group. 


0.75 


Moderate 

positive 


X 


X 






Calhoon & 
Fuchs (2003) 


X 








PALS and CBM show promise for 
improving the computational skills of 
secondary students with disabilities. 
However, since the PALS and CBM 
interventions were implemented in 
combination, it is impossible to attribute 
the outcomes strictly to CBM. 


-0.29 to 
0.40 


Trivial 
negative to 
small 
positive 






X 
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study and 
Design^ 


Target Students 


Summary of Findings 


Effect 

Slze^ 


Effect 

Category^ 


Common Empirical Flaws'* 


K-8 


High 

School 


Developmental 

Education 


College 

Level 


Student Design 
Non-Equivalent 


Teacher Design 
Non-Equivalent 


Possible 

Attrition 

Issues 


No 

Comparison 

Group 


Dees (1991) 






X 




Students in the cooperative learning 
sections performed significantly better on 
the algebra word problem and geometry 
proof-writing sections than students in the 
traditional sections. 


0.39 to 
0.56 


Small 

positive 




X 






DePree 

(1998) 






X 




Latino and female students in cooperative 
learning classes had positive gains in self- 
reported math confidence relative to the 
control group, but there were no 
differences in the achievement gains of 
treatment and control groups. 


0.45 to 
0.72 


Small to 
moderate 
positive 




X 


X 




Duncan & 
Dick (2000) 








X 


The Math Excel students attained 
significantly higher grades than the non- 
Math Excel students, and students in the 
Math Excel Program outperformed their 
predicted grades, as determined by their 
SAT Math score, by half a grade point. 


N/A 


Moderate 

positive 


X 


X 


X 




Fuchs et al. 
(1995) 


X 








Students with disabilities and average- 
achieving students in the PALS groups 
outperformed control group students on 
operations and application tests, while 
low-achieving students outperformed 
their counterparts only on the math 
operations test. 


0.07 to 
0.95 


Trivial to 
moderate 
positive 










Fuchs et al. 
(1997) 


X 








Students with disabilities and low- 
achieving students in the peer mediated 
instruction (PMI) groups outperformed 
their peers in the control group, and when 
tasks that emphasized understanding were 
added to the PMI treatment, both 
average- and high-achieving students 
made greater gains than their 
counterparts in the control group. 


0.18 to 
1.15 


Trivial to 
moderate 
positive 
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study and 
Design^ 


Target Students 


Summary of Findings 


Effect 

Size^ 


Effect 

Category^ 


Common Empirical Flaws'* 


K-8 


High 

School 


Developmental 

Education 


College 

Level 


Student Design 
Non-Equivalent 


Teacher Design 
Non-Equivalent 


Possible 

Attrition 

Issues 


No 

Comparison 

Group 


Fuchs et al. 
(2001) 


X 








Medium- and low-achieving students and 
students with disabilities in the peer- 
assisted learning strategies (PALS) group 
outperformed the control group, but 
growth was higher for high-achieving 
control group students than high- 
achieving PALS students. 


-0.20 to 
0.53 


Small 

negative to 
small 
positive 










Ginsburg- 
Block & 
Fantuzzo 
(1998) 


X 








Peer collaboration methods had a positive 
impact on the computational and word 
problem skills, academic motivation, and 
self-concept of third- and fourth-grade 
students. (Peer collaboration and problem 
solving methods were not significantly 
more effective in combination than 
implemented separately.) 


0.29 to 
0.36 


Small 

positive 










Flooker 

(2010) 






X 




Students in pre-algebra classes that used 
peer-led team leader (PLTL) workshops 
had higher persistence and completion 
rates than control group students. 


N/A 


Trivial to 
small 
positive 


X 


X 


X 




Karper & 

Melnick 

(1993) 


X 








There were no significant differences 
between students using Team Accelerated 
Instruction and students in comparison 
classrooms at any grade level on scores of 
math aptitude, concepts, and 
computations. 


0.00 


Trivial 

positive 










Keynes & 
Olson (2000) 

Descriptive 








X 


Students in the Calculus Initiative 
classrooms had higher GPAs, pass rates, 
and retention rates than students in 
traditional calculus. 


N/A 


Small 

positive 


X 


X 
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study and 
Design^ 


Target Students 


Summary of Findings 


Effect 

Size^ 


Effect 

Category^ 


Common Empirical Flaws'* 


K-8 


High 

School 


Developmental 

Education 


College 

Level 


Student Design 
Non-Equivalent 


Teacher Design 
Non-Equivalent 


Possible 

Attrition 

Issues 


No 

Comparison 

Group 


Norwood 

(1995) 






X 




Compared to students who took 
Developmental Algebra in the semester 
when instructors used traditional 
methods, a higher proportion of students 
who took Developmental Algebra in the 
semester that the instructors used the 
Learning Model of cooperative learning 
completed their first college-level math 
course. 


N/A 


Small 

positive 


X 








Summers & 

Svinicki 

(2007) 








X 


Students in the cooperative learning 
classrooms reported significantly more 
motivation for mastery and perceived 
more interactive learning and classroom 
community but reported significantly less 
performance motivation than students in 
the traditional classrooms. 


-0.65 to 
0.37 


Moderate 
negative to 
small 
positive 


X 


X 


X 




Metacognition 


Abdalkhani 
& Menon 
(1998) 






X 


X 


In earlier courses in which journal writing 
was not used in math, students had a 
mean score of 65% on quizzes, while 
studends in the course that incorporated 
journal writing had a mean quiz score of 
72%. 


N/A 


Trivial 

positive 


X 


X 






Hiebert & 

Wearne 

(1993) 


X 








The opportunity for students to explain, 
describe, and question their learning 
contributed to higher gains in 
achievement in classrooms that used 
classroom discourse versus classrooms 
that did not. 


0.5 to 
1.5 


Moderate to 
large 
positive 




X 
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study and 
Design^ 


Target Students 


Summary of Findings 


Effect 

Size^ 


Effect 

Category^ 


Common Empirical Flaws'* 


K-8 


High 

School 


Developmental 

Education 


College 

Level 


Student Design 
Non-Equivalent 


Teacher Design 
Non-Equivalent 


Possible 

Attrition 

Issues 


No 

Comparison 

Group 


Hohn & Frey 
(2002) 


X 








The SOLVED method, which stands for 
"state the problem, options to use, links to 
the past, visual aid, execute your answer, 
and do check back," had a positive impact 
on third-, fourth-, and fifth-grade students' 
math performance compared to control 
group students. 


0.45 to 
0.93 


Small to 
moderate 
positive 




X 






Porter 

(1996) 








X 


There were no differences in the number 
of procedural errors made by students in a 
college calculus course that used writing 
to learn math and students in a 
comparison course on a final exam, but 
students in the treatment group made 
more conceptual errors. 


-0.63 to 
0.09 


Moderate 
negative to 
trivial 
positive 


X 


X 






Pugalee 

(2001) 

Qualitative 




X 






A metacognitive framework emerged in 
high school students' writings about their 
problem-solving processes. 


N/A 


N/A 








X 


Pugalee 

(2004) 

Qualitative 




X 






Writing their problem solving process was 
more beneficial for a group of 20 high 
school math students than verbalizing 
their problem-solving process. 


N/A 


N/A 








X 


Schurter 

(2002) 






X 




Students who received direct instruction 
in the use of comprehension monitoring or 
Polya's four-step problem-solving method 
performed better in mathematical 
problem solving than those who did not. 


N/A 


Small 

positive 




X 






Tournaki 

(2003) 


X 








Students with and without learning 
disabilities who received strategy 
instruction in verbalizing the problem- 
solving process improved much more on 
an addition facts test than students who 
received drill-and-practice instruction. 


0.10 to 
1.58 


Trivial to 
large 
positive 
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study and 
Design^ 


Target Students 


Summary of Findings 


Effect 

Size^ 


Effect 

Category^ 


Common Empirical Flaws'* 


K-8 


High 

School 


Developmental 

Education 


College 

Level 


Student Design 
Non-Equivalent 


Teacher Design 
Non-Equivalent 


Possible 

Attrition 

Issues 


No 

Comparison 

Group 


Problem Representation 


Brenner et 
al. (1997) 


X 








Pre-algebra students who received 
instruction in problem representation 
were better able to create and apply 
multiple representations than control 
students at significant levels, but the 
control students did significantly better on 
a test of symbol-manipulation skills. 


-0.21 to 
0.91 


Small 

negative to 
moderate 
positive 










Chappell 

(2006) 








X 


Students in the concept-based calculus 
sections scored significantly better on the 
midterm and final exams than the 
students in the traditional sections, except 
for on the final procedural skill section. 


0.34 to 
0.64 


Small to 
moderate 
positive 










Fuson & 
Briars (1990) 


X 








The addition and subtraction performance 
of the second graders in learning/teaching 
approach classrooms was above that 
reported for typical third graders. 


N/A 


Very large 
positive 


X 




X 




Jitendra et 
al. (1998) 


X 








Elementary students with or at risk for 
mild learning disabilities who received 
schema-based instruction, direct 
instruction in using schematic diagrams 
and multiple solution strategies, improved 
their word problem performance more 
than students in the control group. 


0.57 to 
0.81 


Small to 
moderate 
positive 










Jitendra et 
al. (2009) 


X 








A diverse group of high- and low-ability 
students who received schema-based 
instruction improved their understanding 
of ratio and proportion more than the 
control group. 


0.45 to 
0.56 


Small 

positive 











55 





study and 
Design^ 


Target Students 


Summary of Findings 


Effect 

Size^ 


Effect 

Category^ 


Common Empirical Flaws'* 


K-8 


High 

School 


Developmental 

Education 


College 

Level 


Student Design 
Non-Equivalent 


Teacher Design 
Non-Equivalent 


Possible 

Attrition 

Issues 


No 

Comparison 

Group 


Silver & 
Stein (1996) 

Descriptive 


X 








Test performance and number of students 
eligible for ninth-grade algebra increased, 
and QUASAR students performed 
significantly better on NAEP compared to 
a similar sample of urban, low-SES 
students. 


N/A 


Small 

positive 








X 


Stein, 
Grover, & 
Henningsen 
(1996) 

Qualitative 


X 








QUASAR students were observed using 
multiple strategies and representations to 
solve and explain their solutions but had 
difficulty maintaining a high level of 
cognitive processing during many of the 
challenging tasks. 


N/A 


N/A 








X 


Witzel et al. 
(2003) 


X 








Students with disabilities and at-risk 
students who received concrete- 
representational-abstract instruction 
showed greater improvements in their 
performance on single- and multiple- 
variable algebra equations than similar 
students receiving traditional instruction. 


0.52 to 
0.87 


Small to 
moderate 
positive 










Zawaiza & 

Gerber 

(1993) 








X 


Community college students with learning 
disabilities who received a schema-based 
intervention made greater gains on a word 
problem test than those who did not, and 
they performed at almost the same level 
as their math-competent peers. 


0.27 to 
1.11 


Small to 
large 
positive 


X 


X 


X 




Application 


Bottge et al. 
(2001) 


X 








The students in the remedial Enhanced 
Anchored Instruction (EAI) class matched 
the performance of students in the pre- 
algebra classes on the problem-solving 
and maintenance test but not on the 
computation test. 


N/A 


Trivial 

positive 


X 


X 
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study and 
Design^ 


Target Students 


Summary of Findings 


Effect 

Size^ 


Effect 

Category^ 


Common Empirical Flaws'* 


K-8 


High 

School 


Developmental 

Education 


College 

Level 


Student Design 
Non-Equivalent 


Teacher Design 
Non-Equivalent 


Possible 

Attrition 

Issues 


No 

Comparison 

Group 


Bottge et al. 
(2002) 


X 








Students in classrooms using EAI 
performed better than the group receiving 
traditional instruction on a video problem 
test but not on a computation and word 
problem test. 


0.11 to 
0.86 


Trivial to 
moderate 
positive 


X 


X 






Brenner 

(1998) 


X 








Native Hawaiian kindergarten students 
exposed to culturally responsive math 
instruction for a year performed better, on 
average, on the standardized math 
assessment than kindergarten students in 
the same class the year before. 


N/A 


Small 


X 








Ellington 

(2005a) 








X 


Students in modeling sections had higher 
levels of self-reported confidence, lower 
levels of anxiety, and lower withdrawal 
rates than students in traditional sections. 


N/A 


Trivial to 
small 
positive 


X 


X 






Ellington 

(2005b) 








X 


Students in modeling sections performed 
significantly better on an assessment and 
had higher pass rates than students In 
traditional sections. 


0.41 


Small 

positive 


X 


X 






Ganter & 
Jiroutek 
(2000) 








X 


Calculus sections that utilized long-term 
projects In the computer lab did not 
perform better than the control sections 
on the final exam. On the standardized 
exam the control group outperformed the 
treatment group. 


N/A 


Trivial 

negative 




X 






Hickey, 
Moore, & 
Pelligrino 
(2001) 


X 








From third to fifth grade, students in the 
reform-oriented Jasper videodisc 
classrooms had the largest gains on the 
problem-solving and conceptual sub-tests 
and the largest decline on the 
computation sub-test, compared to 
students in comparison classrooms and 
non-reform-oriented treatment 
classrooms. 


N/A 


Trivial 
negative to 
trivial 
positive 
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study and 
Design^ 


Target Students 




Effect 

Size^ 


Effect 

Category^ 


Common Empirical Flaws'* 


K-8 


High 

School 


Developmental 

Education 


College 

Level 


Summary of Findings 


Student Design 
Non-Equivalent 


Teacher Design 
Non-Equivalent 


Possible 

Attrition 

Issues 


No 

Comparison 

Group 


Hollar & 

Norwood 

(1999) 








X 


Scores on the function test were higher for 
the graphing-approach group than for the 
control group, but there were no 
significant differences in terms of final 
exam scores (scores not reported). 


1.02 


Large 

positive 


X 


X 






Lipka & 
Adams 
(2004) 


X 








Teachers were randomly assigned to use 
culturally relevant pedagogy or traditional 
instruction with their sixth-grade Yu'pik 
students, and students in the treatment 
classrooms had higher gains in 
understanding the concepts of perimeter 
and area than students in the control 
group. 


0.44 to 
0.63 


Small to 
moderate 
positive 




X 






O'Callaghan 

(1998) 








X 


The Computer-Intensive Algebra students 
scored higher than students in the 
traditional algebra classrooms on the 
function test, and there were no 
significant differences in final exam 
performance (scores not reported). 


0.86 to 
1.07 


Large 

positive 




X 


X 




Vasquez 

(2004) 

Descriptive 






X 




A greater percentage of students passed 
higher level math courses after the 
introduction of the Algorithmic 
Instructional Technique. 


N/A 


Trivial to 
small 
positive 




X 




X 


Understanding Student Thinking 


Adams 

(1997) 

Qualitative 








X 


The introduction of graphing calculators 
into the classroom improved an 
instructor's assessment methods and 
understanding of student learning. 


N/A 


N/A 








X 
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study and 
Design^ 


Target Students 


Summary of Findings 


Effect 

Size^ 


Effect 

Category^ 


Common Empirical Flaws'* 


K-8 


High 

School 


Developmental 

Education 


College 

Level 


Student Design 
Non-Equivalent 


Teacher Design 
Non-Equivalent 


Possible 

Attrition 

Issues 


No 

Comparison 

Group 


Allinder et 
al. (2000) 


X 








Students with disabilities performed 
better on a test of computational ability in 
a classroom where teachers used 
curriculum-based measurement (CBM) 
plus self-monitoring of their instructional 
practices than students of teachers who 
only used CBM, and both treatment 
groups performed better than the control 
group. 


0.35 to 
0.92 


Small to 
moderate 
positive 


X 




X 




Boylan & 

Saxon 

(1998) 

Descriptive 






X 




Institutions with exceptional 
developmental education programs based 
on observation and developmental 
education pass rates reported using 
frequent testing in their developmental 
education classrooms. 


N/A 


N/A 








X 


Calhoon & 
Fuchs (2003) 


X 








PALS and CBM show promise for 
improving the computational skills of 
secondary students with disabilities. 
However, since the PALS and CBM 
interventions were implemented in 
combination, it is impossible to attribute 
the outcomes strictly to CBM. 


-0.29 to 
0.40 


Trivial 
negative to 
small 
positive 






X 




Fabry et al. 
(1997) 

Qualitative 






X 




Students self-reported that the classroom 
assessment technique improved their 
perceptions and attitudes about learning. 


N/A 


N/A 








X 


Fuchs & 
Fuchs (1990) 


X 








Students with disabilities who had 
teachers who used CBM and skills analysis 
(which allows teachers to analyze 
proficiency in specific skills) performed 
somewhat better on a computation test 
than students with disabilities whose 
teachers used only CBM, and both 
treatment groups outperformed the 
control group. 


0.28 to 
0.67 


Small 

positive 
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study and 
Design^ 


Target Students 


Summary of Findings 


Effect 

Size^ 


Effect 

Category^ 


Common Empirical Flaws'* 


K-8 


High 

School 


Developmental 

Education 


College 

Level 


Student Design 
Non-Equivalent 


Teacher Design 
Non-Equivalent 


Possible 

Attrition 

Issues 


No 

Comparison 

Group 


Fuchs et al. 
(1991) 


X 








Students with disabilities who had 
teachers who used CBM with expert 
instructional recommendations made 
greater gains on an operations test than 
students in the control group. 


0.84 to 
0.94 


Moderate 

positive 










Fuchs et al. 
(1994) 


X 








Students with disabilities who had 
teachers who used CBM with instructional 
recommendations made slightly greater 
gains on an operations test than students 
with disabilities whose teachers used only 
CBM, and both treatment groups 
outperformed the control group. 


0.16 to 
0.43 


Trivial to 
small 
positive 










Nunnery & 
Ross (2007) 


X 








Students in Accelerated Math (AM) 
classrooms did significantly better on 
standardized assessments than students in 
comparison classrooms. 


0.17 to 
0.22 


Trivial to 
small 
positive 




X 






Siadat et al. 
(2008) 






X 


X 


Students in the Keystone Method classes 
had higher final exam scores and 
persistence rates than students in control 
classes. 


N/A 


Small to 
large 
positive 


X 


X 






Villasenor & 

Kepner 

(1993) 


X 








Students in the cognitively guided 
instruction classrooms outperformed 
students in the control classrooms. 


3.55 to 
5.44 


Very large 
to nearly 
perfect 
positive 


X 








Ysseldyke & 
Bolt (2007) 


X 








AM classrooms performed significantly 
better on the STAR Math test compared to 
control classrooms, but there were no 
significant differences between the two 
groups in Terra Nova performance. 


0.37 


Small 

positive 






X 





60 





study and 
Design^ 


Target Students 


Summary of Findings 


Effect 

Size^ 


Effect 

Category^ 


Common Empirical Flaws'* 


K-8 


High 

School 


Developmental 

Education 


College 

Level 


Student Design 
Non-Equivalent 


Teacher Design 
Non-Equivalent 


Possible 

Attrition 

Issues 


No 

Comparison 

Group 


Ysseldyke & 

Tardrew 

(2007) 


X 


X 






AM third- to sixth-grade classrooms 
performed significantly better on 
standardized assessments than 
comparison classrooms. There were no 
significant findings for seventh- to tenth- 
grade classrooms. 


0.18 to 
0.57 


Trivial to 
small 
positive 




X 






Computer-Based Learning 


Garcia 

(2003) 






X 




Pre- to post-test mean ACCUPLACER 
scores increased for students in an 
elementary algebra class with learning 
style and attitude surveys, workshops, 
computer-based instruction, self- 
assessment, and supplemental instruction. 


N/A 


Small 

positive 






X 


X 


Hagerty & 
Smith (2005) 








X 


The students using ALEKS had higher gains 
from pre- to post-test than the students in 
the traditional classrooms. 


0.49 


Small 

positive 






X 




Lovett et al. 
(2008) 








X 


Students randomly assigned to an eight- 
week accelerated OLI-Statistics hybrid 
course (in which students met with the 
instructor to review and reinforce 
material) performed better on the final 
exam than students randomly assigned to 
the traditional 15-week statistics course. 


N/A 


Small 

positive 


X 


X 






McClendon 
& McArdle 
(2002) 






X 




Retention was higher in the lecture mode 
of instruction versus ALEKS and Academic 
Systems. 


N/A 


Trivial to 
moderate 
negative 


X 


X 


X 




O'Dwyer et 
al. (2007) 


X 


X 






Algebra students in eighth and ninth grade 
using a hybrid online learning model 
scored slightly higher on an algebra test at 
the end of the year. 


0.13 


Trivial 

Positive 






X 
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study and 
Design^ 


Target Students 


Summary of Findings 


Effect 

Size^ 


Effect 

Category^ 


Common Empirical Flaws'* 


K-8 


High 

School 


Developmental 

Education 


College 

Level 


Student Design 
Non-Equivalent 


Teacher Design 
Non-Equivalent 


Possible 

Attrition 

Issues 


No 

Comparison 

Group 


Speckler 

(2008) 

Descriptive 






X 


X 


At 18 colleges, retention and pass rates, 
course enrollments, and/or grades were 
generally higher for students in sections 
that used MyMathLab or MathXL, with a 
few exceptions. 


N/A 


Small to 
moderate 
positive 








X 


Squires et 
al. (2009) 

Descriptive 






X 




At Cleveland State Community College, 
after the Introduction of a course redesign 
model, course completion rates In 
developmental math and subsequent 
college-level math courses increased. 


N/A 


Small 








X 


Stillson & 
Alsup (2003) 

Qualitative 








X 


A higher percentage of students failed the 
course in the semester that ALEKS was 
introduced than in previous semesters. 
Students in the study reported that ALEKS 
in combination with group work, the 
lectures, and individual assistance from 
the instructor was helpful In learning 
algebra. 


N/A 


Trivial to 
small 
negative 


X 


X 






Taylor 

(2008) 






X 


X 


The control group made larger gains from 
pre- to post-test than the ALEKS group on 
the algebra test, but self-reported math 
anxiety decreased more for the ALEKS 
group than for the control group, and self- 
reported attitudes about math improved 
for the ALEKS group and worsened for the 
control group. 


-0.21 to 
-0.12 


Small 

negative 


X 


X 






Twigg 

(2005) 

Qualitative/ 

Descriptive 






X 


X 


Different outcomes were examined at 30 
selected institutions, with in-depth case 
studies on 15 institutions. Colleges that 
redesigned their math courses reported 
increases in retention, math learning, and 
course pass rates and decreased cost per 
student. 


N/A 


N/A 








X 
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study and 
Design^ 


Target Students 




Effect 

Size^ 


Effect 

Category^ 


Common Empirical Flaws'* 


K-8 


High 

School 


Developmental 

Education 


College 

Level 


Summary of Findings 


Student Design 
Non-Equivalent 


Teacher Design 
Non-Equivalent 


Possible 

Attrition 

Issues 


No 

Comparison 

Group 


Waycaster 

(2001) 

Qualitative/ 

Descriptive 






X 


X 


The success rates in 15 developmental 
math courses were not related to the 
method of Instruction (lecture or 
Computer-Aided Instruction). 


N/A 


Trivial 

negative 


X 


X 






Zavarella & 

Ignash 

(2009) 






X 




20% of students In the lecture-based 
course, 42% of students In the hybrid 
course, and 39% of students in the online 
course withdrew. 


N/A 


Small 

negative 


X 


X 






Zhu & 

Polianskaia 

(2007) 






X 




Over most years in a ten year period, a 
higher percentage of students in lecture 
courses had higher pass rates, course 
completion rates, and final exam scores 
than students in computer -mediated 
courses. 


N/A 


Trivial to 
small 
negative 


X 


X 







Notes 1-4 appear on the next page. 
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Table 1 Notes 



1. Study and Design 

Unless otherwise noted, the study is a quantitative study that used a quasi- 
experimental or randomized design. Descriptive studies usually did not utilize 
comparison groups but reported frequency outcomes of students in existing programs. 
Qualitative studies usually involved observation of students in some program or 
instructional environment, and most did not utilize a comparison group. 

2. Effect Size 

Effect sizes are used to compare results across studies with different outcome 
measures and are also important in measuring meaningful changes in outcomes that may 
not necessarily be statistically significant. The table reports Cohen’s d effect sizes, which 
can be interpreted as the standardized difference between the treatment and control group 
means. A positive d indicates that the treatment group had superior outcomes; a negative 
d indicates that the comparison group had superior outcomes. When d was not reported in 
a given paper, the effect size was calculated using means and standard deviations for the 
treatment and control groups as provided in the article, using the following formula, 
where x is the mean, n is the sample size, s is the standard deviation, and the subscripts t 
and c denote treatment and control: 




If the means and/or standard deviations were not provided but the E-statistic was, 
Cohen’s d was calculated using F and the sample sizes of the treatment and control 
groups: 
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3. Effect Category 

For articles that do not report the information to necessary calculate Cohen’s d, 
“N/A” is written in the effect size column. However, some of these articles report group 
percentages or correlations that allow for an estimation of their effect size category. 
Hopkins (2009) proposes a scale that can be used to compare traditional effect size 
estimates to differences in percentages. The following scale was used to compare effect 
size estimates across various types of effects, in order to categorize them by size: 



Measure 


Trivial 


Small 


Moderate 


Large 


Very Large 


Nearly Perfect 


Correlation coefficient (r) 


0-0.1 


0. 1-0.3 


0. 3-0.5 


0.5-0.7 


0.7-0. 9 


0.9-1.0 


Standardized difference in means (d) 


0-0.2 


0.2-0.6 


0.6-1. 2 


1. 2-2.0 


2.0-4.0 


4.0-°° 


Percentage difference 


0-10 


10-30 


30-50 


50-70 


70-90 


90-100 



4. Common Empirical Flaws 

Four common methodological flaws were identified across the studies. Studies 
that did not have any of these flaws are considered rigorous. Below is an explanation of 
each empirical flaw. 

Student design non-equivalent. The most common weakness, especially of the 
quantitative studies whose target population is developmental or college-level students, is 
a non-equivalent student design. Most of these studies allowed students to self-select into 
treatment and control groups but neglected to collect pre-treatment ability and 
demographic information on the student participants. As a result, any significant 
differences in outcomes could be due to the preexisting differences between the treatment 
and control students rather than the instructional intervention. Another common research 
design is to compare the outcomes of students who received the treatment to the 
outcomes of students from previous years or semesters, when traditional instructional 
practices were in use. However, this comparison introduces time-varying characteristics 
that are not controlled for and, therefore, could explain any differences in outcomes. 

Other studies do collect pretreatment information but do not use it to conduct a rigorous 
analysis of group equivalence. For example, these studies do not adjust post-treatment 
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test performance for pre-treatment test performance, or they do not test for statistical 
differences between treatment and control group pre-test scores. Finally, some articles do 
not report pre-test scores, but the author claims that they revealed no significant 
differences between the treatment and control groups. This lack of transparency raises 
doubts about the authors’ claims and the internal validity of these studies. 

Teacher design non-equivalent. The second flaw, teacher design non- 
equivalency, could arise from a number of different study features. First, most of the 
reviewed empirical studies do not describe how teachers were assigned to the treatment 
and comparison groups and do not report characteristics of teachers who volunteered for 
the treatment and control sections. These studies disregard the influence that the 
underlying characteristics of individual teachers have on educational outcomes; 
variability in teacher characteristics could confound the relationship between the 
intervention and any outcomes. Second, even in a few randomized and quasi- 
experimental studies, treatment teachers demonstrated intrinsic motivation to improve 
their teaching by volunteering for the treatment group. That they possessed intrinsic 
motivation can be inferred from the fact that the treatment required them to attend 
training. Therefore, differences in instructional quality or motivation may be responsible 
for the outcomes of the study rather than the treatment. Finally, in a number of studies, 
the researcher was also the instructor of the treatment and/or control group(s), implying 
an unconscious or maybe even a conscious investment in ensuring the classroom 
intervention is effective, which calls into question the researcher’s impartiality in the 
implementation of the treatment and analysis of the results. 

Possible attrition issues. Many studies do not address the substantial attrition that 
occurred over the course of the study. High attrition is common in studies at the 
developmental education level, where course dropout rates are high. If the attrition is 
more pronounced for students in a treatment or control group, it could bias the results of 
the study. For example, if lower-performing students were more likely to drop out of the 
control group, then this could have biased downward the impact of the treatment, but if 
lower-performing students dropped out of the treatment group at higher rates, then this 
could have inflated the impact of the treatment (for an illustration of differential attrition, 
see Figure 1 in Jaggars [2011]). 
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No comparison group. A comparison group provides information about what 
would have happened if the students did not receive the instructional intervention. It is 
not possible to attribute any observed outcomes to the instructional intervention when it 
is unknown if the student outcomes would have been similar, better, or worse in the 
absence of the treatment. Most of the quantitative studies employ a comparison group, 
while most of qualitative and descriptive studies do not. These studies provide important 
descriptive information about the outcomes of an instructional intervention or qualitative 
data about the challenges individual students may face when using alternative or reform- 
based math pedagogy, but they do not provide evidence of a causal link between the 
instructional practice or program and student outcomes. 
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